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Definition:  A  solution  x  e  F  is  a  local  optimal  solution  for 
NLP  if  tnere  exists  a  neighborhood  N  of  x   such  that  f(x  )  <;  f(x) 
for  all   x  t;  F  n  N. 

If  f,  g.  are  all  convex  functions,  then  we  call  NLP  a  convex 
program,  and  the  following  properties  hold: 

1.  The  feasible  region  F  is  a  convex  set. 

2.  Any  local  optimal  solution  to  NLP  is  also  a  global 
optimal  solution. 

For  general,  possibly  nonconvex  f,  g.  we  can  no  longer  guarantee 
these  properties.  In  particular, 

1.  The  feasible  region  is  not  necessarily  a  convex  set,  in  fact 
it  is  not  even  necessarily  a  connected  set. 

2.  There  may  exist  local  optimal  solutions  which  are  not  global 
optimal . 

Most  of  the  well  known  nonlinear  optimization  algorithms  will  terminate 
when  a  local  solution  has  been  found  and  are  not  able  to  determine  whether 
or  not  this  solution  is  also  global.  Algorithms  for  extending  local  opti- 
mizers to  find  global  solutions  have  recently  been  developed  (see  [9] 
for  a  summary  of  several  methods).  They  can  conveniently  be  divided  into 
two  classes,  heuristic  methods  and  algorithmic  methods. 

Algorithmic  methods  are  guaranteed  to  converge  to  a  global  optimal 
solution  (perhaps  only  as  the  limit  of  an  infinite  sequence  of  operations). 
Also,  and  perhaps  more  important,  the  computations  are  organized  in  such 
a  way  that  when  the  global  solution  is  found,  the  algorithm  can  eventually 
recognize  that  tne  solution  is,  in  fact,  global  and  will  thus  terminate. 
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The  best  examples  of  algorithmic  methods  are  the  various  versions  of  the 
oranch  and  bound  algorithm  as  developed  by  Falk  ana  Soland[2],  Falk  [3], 
Soland  [11],  and  Beale  &  Tomlin  [1]. 

Heuristic  methods,  on  the  other  hand,  can  be  characterized  as 
techniques  which  are  plausible  approaches  to  global  optimization,  but  for 
which  no  convergence  proof  is  known.  Many  heuristics  can  guarantee 
arDitrarily  high  probabilities  of  attaining  the  global  optimum  (if  sufficiently 
large  numbers  of  computations  are  done),  but  the  critical  difference  is  that 
a  heuristic  can  never  proclaim  with  certainty  that  the  global  solution  has 
been  found.  Some  examples  of  heuristic  methods  are  found  in  Hesse  [7], 
Hartman  [5,6]  and  Opacic  [10]. 

Algorithmic  methods  concentrate  on  gathering  the  information  needed 
to  prove  that  the  best  solution  found  is,  in  fact,  global,  while  heuristic 
methods  concentrate  on  rapidly  attaining  good  solutions  and  perhaps  some 
degree  of  confidence  that  the  best  solution  found  is  a  global  optimal  solu- 
tion. Particularly  relevant  to  this  last  goal  is  the  work  by  Liau,  Hartley, 
and  Sielken  [8]  in  which  statistical  confidence  regions  for  the  global 
solution  are  developed. 

Algorithmic  methods  are  known  only  for  certain  classes  of  problems. 
In  particular  the  branch  and  bound  methods  mentioned  above  require  that  NLP 
be  a  separable  program.  Heuristics  are,  in  general,  not  so  restricted, 
although  particular  computational  schemes  may  apply  only  to  particular 
problem  types. 

The  primary  advantage  of  algorithmic  methods  over  heuristics  lies 
in  the  positive  guarantee  of  global  optimality  which  algorithmic  methods 
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can  provide.  The  disadvantages  are  first,  that  algorithmic  methods  may 
be  too  slow  to  converge  so  that,  in  practice,  the  guarantee  of  global 
optimal ity  is  meaningless;  and  second,  that  special  problem  structures 
are   required.  The  preferred  method  probably  depends  on  the  particular 
problem  to  be  solved.  There  is  not  sufficient  computational  experience 
available  at  this  time  to  formulate  general  rules  of  choice. 
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1 1 .  COMPONENTS  FOR  HEURISTIC  GLOBAL  OPTIMIZERS 

In  this  section  we  discuss  some  basic  building  blocks  which  can  be 
combined  in  various  ways  to  develop  heuristic  global  optimizers  for  solving 
the  nonconvex  NLP  problem.   In  particular,  the  following  structural  components 
wi 1 1  be  considered. 

A)  Search  methods  for  finding  constrained  local  solutions  to 
NLP. 

B)  Procedures  for  obtaining  new  starting  points  for  repeated 
application  of  the  local  search  methods. 

C)  Procedures  for  moving  away  from  one  local  solution  in  an 
attempt  to  find  a  better  one. 

D)  Procedures  for  early  termination  of  local  searches  which 
appear  to  be  converging  to  local  solutions  which  are  already 
known. 

Combination  of  these  components  to  form  heuristic  global  optimizers  will  be 
considered  in  section  III. 

A.  Local  Optimizers.  Methods  for  finding  constrained  local 
optimal  solutions  to  nonlinear  programs  are  well  developed.   In  particular 
penalty  function  methods  would  seem  particularly  relevant  to  the  purpose  of 
nonconvex  optimization  since  they  do  not  depend  upon  any  convexity  properties 
of  the  constraint  functions  or  feasible  region.  We  distinguish  two  classes 
of  methods: 

A_L   Interior  penalty  functions  (also  known  as  barrier 
functions)  the  best  known  example  is  the  SUMT  function  [4]. 
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m  , 

P(x,r)       f(x)   +   r   J       ■  A-(  }    ,     r  >  0 

1=1  1      ^1 V       ' 


Interior  methods   require  an  initial    feasible     x. 

A2.     Exterior  penalty  functions   such  as 

m  ? 

Q(x,r)   =  f(x)   +  r     y     [min(0,  b.-g.(x))r 

i=l  ]      1 

Exterior  methods  do  not  require  initial  feasible  solutions.   It  should  be  noted 
however,  that  for  nonconvex  problems  there  is  no  guarantee  that  the  exterior 
penalty  function  method  will  converge  to  a  feasible  solution,  so  we  must  be 
prepared  to  correct  for  this  problem  if  it  occurs. 

B.  Starting^  Points.  Perhaps  the  oldest  proposal  for  global 
optimization  is  to  repeatedly  apply  a  local  optimizer  starting  each  local 
optimization  from  a  different  initial  point.  Comparisons  of  several  proce- 
dures for  effectively  selecting  starting  points  for  unconstrained  global 
optimization  can  be  found  in  [5]  and  [6],  and  similar  procedures  are  worthy 
of  consideration  for  the  constrained  NLP.  The  constrained  case  is  more 
difficult  since  if  we  wish  to  use  an  interior  penalty  method  for  the  local 
optimizer,  then  the  starting  point  must  be  in  the  interior  of  the  feasible 
region  F.  Several  possible  methods  are  indicated  below: 

Bl .  Sample  x  from  a  uniform  random  distribution  over  the 
set 


S  =  {x  |  L  .  <  x.  <  U  . ;     j  =  1  ,  . . .  ,  n} 

J       J       J 


If  also 


x  e  R  =  {x  |  g.(x)  <  b . ;   i  =  1,  ...  ,  m} 
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then  use  x  as  a  new  starting  point  for  local  optimization.  Otherwise 
repeat  the  process,  continuing  to  sample  x  e  S  until  x  l  R   is 
found.  The  probability  of  finding  a  feasible  starting  point  at  any  trial 
is  the  ratio  of  the  volume  of  S  n  R   to  the  volume  of  S   which  is 
generally  not  known.  Note,  however,  that  if  the  problem  contains  an 
equality  constraint,  then  the  volume  of  R   is  zero  and  this  method  is 


not  appropriate, 


B2.  Sample  x  c   S   randomly.   If  also  x  c  R   then  x 


is  a  new  starting  point.   If  x  £  R  ,  let 


11  =  {i  |  g. (x)  £  b, ,  i  =  1 ,  . . .  ,  m} 
and 

12  =  {i  |  g.j(x)  <  b. ,  i  =  1 ,  ...  ,  m} 

Then  formulate  the  "feasibility  finding  problem" 


FFP   min  I        g,(x) 

i  e  II 


st.  g^x)  <;  bi   i  e 


12 


and  use  an  interior  penalty  function  method  to  solve  it.  If  a  violated 
constraint  (in  II)  becomes  satisfied  during  the  solution  process  it  is 
moved  to  12  and  the  process  continues  until  either  a  feasible  starting 
point  is  found,  or  a  minimum  to  FFP  is  found  but  is  not  in  R  .   In  the 
latter  case  a  new  x  e  S   is  sampled  and  the  process  repeats.  This 
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feasibility  finding  problem  is  due  to  Fiacco  &  McCormick  [4]. 

B3.  If  exterior  penalty  functions  are  to  be  used  for  NLP, 
then  any  x  c  E   is  suitable  as  a  starting  point,  but  it  would  seem 
reasonaDle  to  restrict  the  choice  to  a  random  selection  of  x  e  S.  Un- 
fortunately, there  is  no  guarantee  that  convergence  to  a  feasible  point 
will  occur  since  the  exterior  penalty  function  may  be  unbounded  over  E  . 
Since  exterior  penalty  functions  generally  converge  to  a  point  which  is 
slightly  infeasible  it  may  be  difficult  to  decide  when  the  pathological 
behavior  is  occurring. 

B4.  A  better  use  of  exterior  penalties  might  be  in  a 
mixed  interior  -  exterior  function.  Select  x  e  S   randomly.  Let  II 
and  12  be  defined  as  above,  and  formulate  the  mixed  interior  -  exterior 
penalty  function  problem: 


*jn  f(x)  ♦  r   J      &;:lTx7  +  r  I  [nnn(0,b  -g.(x)]2  +  r  [  (^  ♦  ^ 

cs  En  icI2       i    yiv  lCll  J  =  l       j       J  J       J 


Minimization  of  this  penalty  function  will  maintain  feasibility  for  the 

constraints  in  12  and  the  upper  and  lower  bounds  on  x   while  attempting  to 

J 

converge  to  feasibility  for  the  violated  constraints  in  II. 

B5.  Each  of  the  above  methods  Bl  -  B4  selects  random  points 
x  e  S   without  regard  to  where  previous  random  points  were  selected.   In  [5] 
a  method  is  developed  which  keeps  track  of  the  previously  selected  starting 
points  and  the  trajectories  from  these  starting  points  to  the  local  optimal 
solutions  found  during  previous  searches.  The  new  x  r  S   can  then  be 
selected  to  avoid  researching  portions  of  S  which  have  already  been  searched 
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This  modification  could  be  applied  to  any  of  Bl  -  B4  as  presented  above. 
Evidence  in  [5]  for  the  unconstrained  problem  indicates  that  this  modification 
may  significantly  improve  the  efficiency  of  global  search  procedures. 
Details  of  the  procedure  can  be  found  in  [5]. 

C.  Leaving  a  Local  Solution.  Once  a  local  solution  has  been  found 
by  the  local  search  algorithm,  we  face  the  problem  of  how  to  continue  the 
search  to  find  another  hopefully  better  local  solution.  One  procedure  is 
to  select  a  new  starting  point  as  in  Bl  -  B5  hoping  that  a  local  search 
from  the  new  point  will  converge  to  a  new  local  solution. 

A  more  systematic  procedure  is  to  move  away  from  the  local  solution 
keeping  track  of  the  penalty  function  value.   Initially  this  value  will 
increase,  (since  we  are  leaving  a  local  minimum).   If  it  subsequently  begins 
to  decrease  then  the  chances  are  good  that  we  have  moved  into  tne  "region 
of  attraction"  of  a  different  local  minimum.  Starting  the  local  optimiza- 
tion algorithm  at  this  point  should  then  lead  to  a  new  local  optimal  solu- 
tion to  NLP.  Two  versions  of  this  idea  are: 

Cl_.  Hove  away  from  the  local  solution  along  some  feasible 
direction  d  (one  which  points  into  the  feasible  region  F  ).  This 
involves  potential  difficulties  if  the  boundary  of  F  is  reached  before 
the  penalty  function  value  decreases.  Perhaps  a  better  idea  is  the  following: 

C2.  Suppose  P(x,r)  is  the  penalty  function  (interior, 
exterior,  or  mixed)  which  has  been  minimized  to  find  the  current  local 
solution  x  .  Add  a  new  constraint 


*  ■  2    2 


q  (x)  =  T  (x  .-x  . )  ^  bri 
yov  '        j^-j  v  J  Jy     0 
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to  the  NLP  problem.  This  constraint  defines  3  hypersphere  of  radius 
t>n  centered  at  x   whose  interior  now  becomes  infeasible,  hence 
excluding  trie  local  solution  x  .  Let 

*  2 

P  (x,r,bQ)  =  P(x,r)  +  r  [min(0,g0(x)-bQ)] 

be  the  penalty  function  resulting  from  augmenting  the  original  P(x,r) 
witn  an  exterior  term  for  the  new  constraint.  Let  x*(b~)  be  the  solution 

JL. 

to  min  P  (x,r,bn).  Gradually  increasing  the  radius  b   of  the  exclusion 

x 
constraint  will  tend  to  drive  x  (b~)  away  from  the  local  solution  x  . 

Continue  to  increase  bn  until  either, 

a)  P(x*(bn),r)  increases  and  then  eventually  decreases.  Then 
continue  to  minimize  P(x,r)  starting  from  this  new  point,  or 

b)  no  feasible  x  (b-J  can  be  found.  Then  procedure  C?  has 
failed. 

Hyperspheric  exclusion  constraints  have  been  previously  considered 
by  otners  (notably  Hesse  [7])  as  devices  for  excluding  local  solutions. 
In  these  cases,  nowever,  the  exclusion  constraints  have  been  retained 
for  tne  subsequent  local  searches  thus  permanently  excluding  the  local 
solution  x   from  feasibility  in  the  subsequent  searches.  This 
necessitates  searching  for  the  smallest  possible  radius  b^  to  avoid 
the  chance  of  also  inadvertently  excluding  the  global  solution  from  further 
consideration.  For  our  purposes  the  exclusion  constraints  are  only  temporary 
devices  for  moving  away  from  a  local  solution.  The  risk  of  subsequently 
revisiting  this  same  local  solution  is  deemed  preferable  to  the  risk  of 
permanently  excluding  the  global  solution. 
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D.  Termination _of  LocaJ  Searches.   If  procedures  such  as  those 
referenced  in  B5  are  used  to  keep  track  of  previously  found  local 
minima  and,  perhaps,  the  search  trajectories  leading  to  them,  then 
subsequent  local  searches  can  be  terminated  prior  to  convergence  if 
they  seem  to  be  repeating  work  which  has  already  been  done. 

Dl .  The  simplest  procedure  is  to  maintain  a  record  of 
the  local  minima  found  so  far.  If  a  subsequent  search  comes  within  a 
tolerance  distance  6  of  one  of  these  solutions,  then  terminate  that 
search.  This  procedure  can  save  considerable  effort  even  if  6  is 
quite  small  since  the  bulk  of  the  work  with  most  local  searches  seems 
to  be  expended  obtaining  convergence  close  to  the  solution  rather 
than  arriving  in  the  vicinity  of  the  solution. 

D2.  If  also  a  record  of  past  search  trajectories  is 
maintained,  then  we  can  terminate  any  subsequent  search  which  comes 
closer  than  a  tolerance  e  to  one  of  these  trajectories.  This  involves 
substantially  more  bookkeeping  and  testing  than  Dl ,  but  computational 
experience  for  the  unconstrained  problem  in  [5]  indicates  that  it  can  be 
a  desirable  procedure. 
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II I .  A_N  OUTLINE  FOR  SOME  HEURISTIC  GLOBAL  OPTIMIZATION  METHODS . 

The  components  described  in  the  previous  section  can  be  combined  to 
form  heuristic  global  optimization  methods.  One  possible  combination 
is  indicated  in  the  flow  chart  of  figure  1. 

Specifying  the  procedure  to  be  used  in  each  block  (e.g.,  Bl ,  B2,  B3, 
B4,  or  B5)  will  determine  a  specific  optimization  method  which  fits  this 
general  pattern.  The  best  choice  is  probably  problem  dependent  in  a 
fashion  which  is  not  now  understood.  We  hope  to  do  further  computational 
experiments  in  the  future  to  improve  our  ability  to  specify  the  particular 
global  search  heuristic  which  is  most  likely  to  be  successful  for  a  given 
problem. 
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FIGURE  1 


Flow  Chart  for  Heuristic  Global  Optimizer 
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